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ABSTRACT 


Learning analytic models are built upon traces students leave in technology-enhanced learning platforms as the digital 
footprints of their learning processes. Learning analytics uses these traces of learning engagement to predict performance 
and provide learning feedback to students and teachers when these predictions signal the risk of failing a course, or even 
dropping-out. But not all of these trace variables act as stable and reliable predictors of course performance. In previous 
research, the authors concluded that trace variables of product type, such as mastery, do a better job than trace variables 
of process type, such as clicks or time-on-task, in predicting performance. In this study, we extend this analysis by 
focusing on learning gains rather than learning outcomes as the most important performance dimension. Distinguishing 
two different levels of initial proficiency, our empirical analysis into the learning of mathematics by first-year university 
students indicates that the lack of stability of the engagement types of process type is mainly explained by learning 
pattern found in students of high initial proficiency. For these students, high levels of engagement lead to lower, rather 
than higher, predicted learning outcomes. Amongst students with lower initial proficiency, higher levels of engagement 
play a different role. 
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1. INTRODUCTION 


Learning Analytics (LA) has the prime aim to improve the study success of students by collecting data about 
the way students learn, designing models that can extrapolate observed learning behaviours into predictions 
of learning outcomes, and provide learning feedback to students and teachers in case those predictions signal 
a risk of unsuccessful learning, or even drop-out (Ifenthaler, 2015; Ifenthaler, Mah, & Yau, 2019). An 
important source of data applied in predictive modelling is trace data generated by technology-enhanced 
learning environments that represent the digital footprints of the learning of students in these learning tools. 
LA is not without challenges (Ifenthaler et al., 2019): how to define study success, and what trace variables 
to choose to provide reliable predictions of that study success, are amongst such challenges. 

Early applications of LA focussed strongly on signalling the risk of drop-out or more generally: failing 
the course. Failing is determined by course performance measures as expressed in terms of grades. Best 
predictors of performance on final exams that are timely enough to allow for educational interventions are 
typically early, of course, performance measures, such as scores on an intermediate test or quizzes. All of 
these are outcome measures that characterise the level of proficiency of the students, but not necessarily the 
amount of learning taking place in the course. And that is actually what course learning objectives are about: 
to learn new concepts, to increase rather than establish students’ levels of proficiency. 

This growth objective is best represented by the concept of learning gain (Rogaten and Rienties, 2018; 
Rogaten et al., 2018). In this learning gain literature, a rich set of learning gains is distinguished, ranging 
from learning gains of affective type, through behavioural types, to cognitive types. The difficulty in 
applying these concepts is in the measurement. That is especially true for learning gains of affective and 
behavioural types, but even if one limits the focus on learning gains of cognitive type only, as we will do in 
this study, measurement is a challenge. Some empirical studies use indirect measures such as the use of 
self-report surveys by students asking to quantify how much they learned, while other studies applied 
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objective tests in a pre- and post-test design (Rogaten and Rienties, 2018; Rogaten et al., 2018). But where 
the use of objective testing is generally to be preferred above self-report surveys, the measurement of 
cognitive learning gains would require the use of equivalent pre- and post-tests, as in progress testing, which 
is hardly ever done. 

The aim of the current study is to investigate potential differences in typical LA type of models with 
learning outcomes versus learning gains as response variables and trace variables of student engagement as 
predictor variables. Research questions in this context are: how do these prediction models differ between the 
two different response variables? And as a follow-up question: what role do the different trace variables, all 
representing different facets of student engagement, in these prediction models? This last research question 
connects to previous research of the authors (Tempelaar et al., 2015, 2017, 2018a), where we found that some 
measures of engagement, such as time-on-task or the intensity of using worked-out examples, are negatively 
related to course outcomes in multivariate prediction equations, rather than the positive relationship 
hypothesized. Does that outcome hold when we predict learning gain rather than learning outcome, is the 
research question of this contribution, thereby providing an immediate extension of our Tempelaar (2017) 
study? 


2. METHODS 


2.1 Context of the Empirical Study 


This study takes place in a large-scale introductory mathematics and statistics course for first-year 
undergraduate students in a business and economics program in the Netherlands. The educational system is 
best described as ‘blended’ or ‘hybrid’ applying the flipped classroom approach. The main component is 
face-to-face: Problem-Based Learning (PBL), in small groups (14 students), coached by content expert tutors 
(in 78 parallel tutorial groups). Participation in tutorial groups is required. Optional is the online component 
of the blend: the use of the two e-tutorials SOWISO (https://sowiso.nl/) and MyStatLab (MSL) (Nguyen et 
al., 2016; Tempelaar, 2017; Tempelaar et al., 2015; 2017). This design is based on the philosophy of 
student-centred education, placing the responsibility for making educational choices primarily on the student. 
Since most of the learning takes place during self-study outside class through the e-tutorials or other learning 
materials, class time is used to discuss solving advanced problems. Thus, the instructional format shares most 
characteristics of the flipped-classroom design. Using and achieving good scores in the e-tutorial practice 
modes is incentivized by providing bonus points for good performance in quizzes that are taken every two 
weeks and consist of items that are drawn from the same item pools applied in the practising mode. This 
approach was chosen to encourage students with limited prior knowledge to make intensive use of the 
e-tutorials. 

The subject of this study is the full 2018/2019 cohort of students (1208 students in total, 1055 students 
included in this study who have full data records). A large diversity of the student population was present: 
only 20% were educated in the Dutch high school system, against 80% being international students, with 56 
nationalities present. A large share of students was of European nationality, with only 3.9% of students from 
outside Europe. High school systems in Europe differ strongly, most particularly in the teaching of 
mathematics and statistics. Therefore, it is crucial that this introductory module is flexible and allows for 
individual learning paths. Students spend on average 28.3 hours in SOWISO and 25.0 hours in MSL, which 
is 30% to 35% of the available time of 80 hours for learning both topics. This study focuses on learning 
engagement of students in only one of the two e-tutorials, to know the math related SOWISO tool, for several 
reasons: the availability of trace data for several types of engagement (see next section), reliable time-on-task 
data by correction for inactive time, and the availability of prior education data regarding mathematics, but 
not statistics. 
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2.2 Instruments and Procedure 


Both e-tutorial systems SOWISO and MSL follow test-directed learning and practising approach. Each step 
in the learning process is initiated by a question, and students are encouraged to (attempt to) answer each 
question. If a student does not master a question (completely), she/he can either ask for hints to solve the 
problem step-by-step, or ask for a fully worked example. After receiving feedback, a new version of the 
problem loads (parameter based) to allow the student to demonstrate his/her newly acquired mastery. 
Students’ revealed preferences for learning strategies are related to their learning dispositions, as we 
demonstrated in previous research (see Nguyen et al. (2016) and Tempelaar et al. (2017, 2018 a, b) for the 
use of worked-examples in SOWISO, and (Tempelaar, 2017) for the use of worked-examples in MSL). This 
study extends Nguyen et al. (2016) and Tempelaar et al. (2018a, b) by investigating three learning strategies 
in the SOWISO tool: worked examples, and supported and tutored problem-solving. 
Figure | demonstrates the implementation of the alternative learning strategies students can opt for a 
sample exercise: 
e Check: the untutored problem-solving approach, offering only correctness feedback after 
problem-solving; 
e Hint: the tutored problem-solving approach, offering feedback and hints to assist the student in the 
several problem-solving steps; 
e Solution: the worked-examples approach; 
e = Theory: asking for a short explanation of the mathematical principle. 


Visualizing bivariate functions 


Consider the function 


At a particular point [a, | the level curve going through that point Is the unit circle. What Is the value f(a, b)? 
f(a,b) 


Ww Check @ Theory & Solution 9 Hint 


Figure 1. Sample of SOWISO exercise with the options Check, Theory, Solution, and Hint 


Our study combines trace data of the SOWISO e-tutorial with prior-education SIS data and course 
performance data. Clicks in the two e-tutorial systems represent an important part of that trace data, but trace 
data can include more than click data only. Azevedo (Azevedo et al., 2013) distinguishes between trace data 
of product and process type, where click data is part of the category of process data. In this study, we will 
combine both process data, as, e.g. the clicks to initiate the learning supports of Check, Hint, Solution and 
Theory mentioned above, but also product data, as, e.g. mastery in the tool, as discussed below. SOWISO 
reporting options of trace data are very broad, requiring making selections from the data. First, all dynamic 
trace data were aggregated over time, to arrive at static, full course period accounts of trace data. Second, 
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from the large array of trace variables, a selection was made by focusing on process variables most strongly 
connected to alternative learning strategies. 

In total, four trace variables were selected: 

e Mastery in the tool, the proportion of exercises successfully solved as product indicator; 
e Attempts: the total number of attempts of individual exercises; 

e Hints: the total number of Hints called. 

e Solutions: the total number of worked-out examples called. 

The Attempts variable is cumulative: it is the sum of calls for hints, for worked-out examples, and 
untutored problem-solving attempts. The system does not trace the number of clicks for the Check option, but 
that number follows from removing the Solution and Hints activities from the total number of Attempts. 

Pass/fail decisions for the course are based on four course outcome measures: performance in the exam, 
both for mathematics (MathExam) and statistics (StatsExam), and the aggregated performance in the three 
quizzes for both topics: MathQuiz and StatsQuiz. The exam brings 40 credits, against 6 credits for quizzes, 
indicating the limited weight of the quizzes (13%). Because of missing adequate trace and prior education 
data for the statistics part, predictive modelling will be limited to the two mathematical performance 
variables, indicated as Exam and Quiz. 

Prior education measures are based on self-reported accounts of the secondary mathematics program at 
the secondary level. Most countries distinguish three levels of math teaching in upper-level high school: the 
highest level, preparing for science university studies, the intermediate level, preparing for social sciences 
university studies, and the lower level, preparing for humanities. Examples are the levels Hl, SL, and SL 
studies of the International Baccalaureate (IB) program. To be admissible to the school, at least the 
intermediate level is required. That allows describing the level of prior education by an indicator variable, 
labelled as MathMajor, having a value of 1 for students educated in the highest track, and 0 for students 
educated in the intermediate track. Shares of these two tracks are 37.1% and 62.9%, respectively. 


2.3 Learning Gains and Learning Outcomes 


Although learning gain, how much students actually learn by taking a course, is obviously the more 
important aim than the learning outcome, how much students know at the end of the course by combining 
newly learned knowledge with prior knowledge, most empirical studies in learning analytics focus on 
learning outcomes. This is best explained by the simple reason that student assessment in any course brings 
exact measures of learning outcomes in the form of exam and quiz grades, whereas learning gain is defined 
by the growth in knowledge throughout the course, and thus requires the measurement of the starting level. 
Such measures are however difficult to operationalize, and for that reason, many empirical studies in learning 
gains take the step to equate learning gains with learning outcomes. In our context, we collected two 
measures that may qualify as a proxy of the level cognitive understanding at the start of the course: the 
outcome of a diagnostic entry test, and self-reported data on prior mathematics education at the secondary 
level. In a preliminary analysis, we investigated the potential of the diagnostic test score as a measure of 
starting level of proficiency and concluded that it was less fit to play that role. There are two different aspects 
to this misfit. First, the diagnostic test is used as an instrument to inform students how well they are prepared 
to take the course. So there is an issue with coverage: the entry test covers topics at a foundational level, 
rather than topics taught in the course itself. Second: students administer the diagnostic test outside class, 
which gives rise to very different response patterns. Some students do not take the test seriously, especially 
when they know they are well prepared, given their high level of proficiency. Other students, especially at the 
other end of the continuum of preparedness, tend to take the entry test ‘too’ seriously and achieve scores that 
are higher than their ready knowledge. The prior education measure does not share these issues. It is less 
fine-grained in that it distinguishes only two different levels, but the coverage of the advanced level program 
overlaps to large extent topics taught in our course, and there are no issues with the administration. 
Therefore, we opted for the prior education dummy variable to describe two different proficiency levels at 
the start of the course: one for students of the advanced track, one for students of the intermediate track. 

In terms of model design, it is not the initial level of proficiency of the two groups itself that is crucial in 
the learning gain approach, but the difference between these two levels. We estimate that difference using 
multivariate regression models, where learning outcome is the response variable, and trace variables together 
with the prior education dummy variable serve the role of predictor variables. The estimate of the regression 
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coefficient of the prior education dummy represents the difference in prior proficiency between the two 
groups expressed in units of the learning outcome variable. In that context, the learning gain equals the 
difference between learning outcome itself, and the estimated effect of prior education. 


3. RESULTS 


3.1 Learning Outcomes and Traces of Learning 


Correlating course outcomes (Exam and Quiz), track of prior education (the dummy MathMajor) with traces 
of engagement with the e-tutorial, both of product type (Mastery) and of process type (Attempts, Solutions, 
Hints, Time) produces some unexpected outcomes: see Table 1. As expected, Exam and Quiz are strongly 
positively related, and both are moderately positively related to having followed math classes at an advanced 
level in high school. The unexpected outcomes refer to the correlations between Exam, the main course 
outcome variable, and the several trace variables of process type. Except for Time, these correlations are 
non-significant or even negative: Attempts, Solutions, Hints. Correlations between the other course outcome 
variable, Quiz, exhibit a very different pattern: these are all significantly positive. 


Table 1. Bivariate correlations of course outcomes, prior education and traces of learning in the e-tutorial; full sample 


Variable 1. 2. 3. 4. 5. 6. ip 8. 
1. Exam 1.000 

2. Quiz 558°" 1.000 

3. MathMajor .381*** 375°" 1.000 

4. Mastery .219"*" 545°" .078" 1.000 

5. Attempts -.037 356°" -.077" .778°** 1.000 

6. Solutions -.210"™" 108°" -.188"*" 557" 916°" — 1.000 

7. Hints -.007 104°" -.052 217" .166"*" .056 1.000 

8. Time 071" 262°" .009 .423""" 360°" .207°*" 202°" 1.000 


Note: ***p <.001; **p <.01; *p <.05. 


Correlations between the several engagement variables of trace type are all positive or non-significant. 
Number of Attempts and number of Solutions are so strongly related, that they constitute a collinear set. On 
average, students do 745 Attempts, of which 325 are calls for worked examples: the Solutions. This ratio of 
44% of the Attempts being calls for Solutions is rather constant over students, causing a high correlation. 
This affects the regression in the next step of the analysis: including both Attempts and Solutions gives rise to 
high variance inflation factors. All regressions have been run with both Attempts and Solutions as a 
predictor, and in all cases, the fit of the regression model with Solutions is far better than that with Attempts. 
For that reason, the Attempts variable is excluded from all regression models. 

The Hints variable behaves differently: not only is its correlation with Attempts no more than weak, 
students call on average no more than 29 Hints, what is in only 4% of their Attempts. 

The regression equations explaining the two course outcomes variables, Exam and Quiz, out of e-tutorial 
trace variables is provided in Table 2. 


Table 2. Regressions of course outcomes on e-tutorial trace data 


Exam Quiz 
LA trace predictor b SE(b) B b SE(b) B 
Mastery 6.143 0.405 0.526""* 1.556 0.066  0.736""* 
Solutions -0.006 0.000 -0.496*™"  -0.001 0.000 -0.329°™* 
Hints -0.006 0.002 -0.091"*  -0.001 0.000 -0.052™ 
Time -0.000 0.000 -0.044 -0.000 0.000 -0.005 
R? 0.222 0.370 


Note: ***p <.001; **p <.01; *p <.05. 
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Trace variables explain 22% of the variation in the Exam outcome and 37% of the variation in Quiz 
outcome. Since quizzes are taken in the same e-tutorial tool as students practice in, and contain items similar 
to those students encounter in the practice mode, it is not surprising that the Quiz outcome variable is better 
explained by the trace variables, than the Exam outcome variable. Remarkably, all trace variables of process 
type have negative or zero betas. That is: taking the mastery level of the student into account, students who 
call more Solutions, or who call more Hints, are predicted to achieve lower course outcomes, on average. 


3.2 Learning Gains and Traces of Learning 


To operationalize learning gains, an estimate of the cognitive abilities of the students before entering the 
course is required. The best estimate available is the level of prior education in high school: advanced track 
or intermediate track. The indicator variable MathMajor serves that role: it indicates if a student took the 
advanced track (MathMajor=1) or the intermediate track (MathMajor=0). Including the MathMajor variable 
in the regression model implies that regression intercepts and slopes differ between the two groups, based on 
differences in correlations. These group-specific correlations are contained in Table 3, with correlations of 
the students of the MathMajor track above the diagonal, and those of the MathIntermediate track below the 
diagonal. 


Table 3. Bivariate correlations of course outcomes, prior education and traces of learning in the e-tutorial; MathMajor 
sample above the diagonal, MathIntermediate sample below the diagonal 


Variable 1. 2. 3. 4. 5. 6. les 8. 
1. Exam 1.000 557" 205°" —-.048 -.253""" 118" -.037 
2. Quiz 456°" — 1.000 505°" 279°" ~~ -,003 .038 148" 
3. MathMajor 

4. Mastery 214°" 574" 1.000 .783""" 527" .203°™* 434°" 
5. Attempts .008 445°" .808"** 1.000 .890"*" .162**" 355°" 
6. Solutions -.128""* 230°" 618°" 927°" 1.000 .076 218°" 
7. Hints .082* .179""" 931" .165"*" .037 1.000 .206"*" 
8. Time .138"*" 334°" .428"*" 380"*" 224""" 194°" 1.000 


Note: ***p <.001; **p <.01; *p <.05. 


Comparing the first two rows with the first two columns makes clear that large difference exists between 
the two groups. Correlations between e-tutorial trace data and the two course outcomes Exam and Quiz are 
more positive everywhere for the students of the intermediate level track than for advanced level track. These 
differences in correlations show up in different regression equations, as exhibited in Table 4. 


Table 4. Regressions of course outcomes on prior education and e-tutorial trace data 


Exam Quiz 

LA trace predictor b SE(b) B b SE(b) B 
MathMajor 3.269 0.339 0.472°" = 0.533 0.054 0.419°™* 
Mastery 5.560 0.385 0.467°" = 1.463 0.062 0.684°™* 
Solutions -0.005 0.000 -0.388""* -0.001 0.000 — -0.230°** 
Hints 

Time 

Mastery* MathMajor 


Solutions*MathMajor -0.002 0.001 -0.120"* 0.000 0.000 -0.126"™* 
Hints*MathMajor -0.012 0.003 -0.105*™* 
Time*MathMajor -0.000 0.000 -0.096"  -0.000 0.000 —-0.073° 
R? 0.324 0.455 

Note: ***p <.001; **p <.01; *p <.05. 


The regression coefficients of the MathMajor indicator variable are positive, signalling that students from 
the MathMajor group score on average 23% higher in the Exam and 37% higher in the Quiz. Other 
differences between the two groups are visible from the estimates of the interaction terms. The negative 
effects of the trace variables of process type, Solutions, Hints and Time, accounting for the Mastery 
interaction term and the main effects are much stronger for the MathMajor group than for the 
MathIntermediate group. 
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4. DISCUSSION AND CONCLUSION 


The MathMajor variable has a strong effect on the two learning outcomes, both directly, as well as indirectly 
through three of the four interaction terms, as visible from Table 4. The direct effect tells us that students of 
the advanced track score, on average 3.3 points higher in the final exam (on a range of 0 ... 20), and 0.53 
points higher in the quiz (on a range of 0 ... 3), than students of the intermediate math track. That direct 
effect is intuitive: the math topics covered in our course have a strong overlap with the curriculum of the 
advanced track, next to the circumstance, that one can expect the proportion of talented math student to be 
higher in that track, than the intermediate level track. That direct effect is our proxy of the difference between 
learning outcome and learning gain. If we bring that direct effect to the left side of the regression equation, 
the resulting equation describes predicted learning gain as a function of trace variables and interaction terms. 

The interesting part of the regressions in Table 4 is the role of the trace variables. The direct effect of the 
trace variables is statistically non-significant, except for Solutions, telling that when controlling for Mastery 
level and prior education, calling more Solutions predicts lower performance. That is: the student who needs 
more worked-out examples to reach the same mastery level as another student needing less worked-out 
examples, will score less in exam and quiz, on average. There is no significant overall effect for Hints and 
Time. However, there are significant interaction effects for Hints, Time as well as Solutions, all with negative 
betas. All are telling the same story: students of the advanced track who, accounting for their mastery level, 
need more Solutions, Hints or Time, are expected to do less well than students of the advanced track who 
reached the same mastery level using fewer Solutions, Hints or Time. The effects for Hints and Time are 
unique to the students of the advanced track: no similar effect is at work in students of the intermediate track. 
The interaction effect of Solutions related to students of the advanced track, adds to the general effect of 
Solutions that relates all students. 

In our previous research (Tempelaar, 2017; Tempelaar et al., 2017, 2018 a, b), investigating learning 
processes in the same course but with different year classes, we invariably found that trace variables of 
process type derived from learning processes in the e-tutorials play a less unambiguous role in prediction 
models. Their bivariate relationships with course performance variables are generally positive (as in this 
study, except for Solutions). However, in multivariate prediction equations that include the Mastery trace 
variable of product type along with the trace variables of process type, these betas of the process type trace 
variables are generally negative (as in this study). Telling the story that the student who needs a more 
intensive process to reach the same mastery level as another student, is expected to do less well in the exam 
and quiz. Where process that is more intensive means: more Solutions, Hints and or Time. 

In the current study, we can specialize these effects by focusing on learning gains rather than learning 
outcomes. We find different regression models for students of the two tracks of prior education. For students 
educated at the advanced track who learned a substantial part of the course coverage in high school, passing 
the course does not require large learning gains. For these students, high levels of engagement rather than 
low levels of engagement represent a risk factor. They require more worked-out examples; more hints and 
more time on task to reach a certain mastery level signals weaker understanding and predicts lower 
performance. Student educated at the lower, intermediate track display a different pattern. These students 
require a large learning gain to pass the course. Spending more time on task and using more hints to reach a 
certain mastery level represents, in this case, no risk factors; only the frequent use of worked examples does. 

This study is based on learning processes of students in a blended learning context, with a one-sided 
observation of those learning processes: we measure every detail of learning activities in the digital mode, 
but any learning in the face-to-face mode is left unobserved. Since the balance between these two learning 
modes will differ from student to student, this is a clear limitation of the context of this study. 
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